Learning to Rank Answers to Why-Questions

The goal of the current research project is to develop a ques­ tion answering system for answering why-questions (why QA). Our system is a pipeline consisting of an off-the-shelf retrieval module followed by an answer re-ranking module. In this paper, we aim at improving the ranking performance of our system by finding the optimal approach to learning to rank. More specifically, we try to find the optimal ranking function to be applied to the set of candidate answers in the re-ranking module. We experiment with a number of ma­ chine learning algorithms (i.e. genetic algorithms, logistic regression and SVM), with different cost functions. We find that a learning to rank approach using either a re­ gression technique or a genetic algorithm that optimizes for MRR leads to a significant improvement over the TF-IDF baseline. We reach an MRR of 0.341 with a success@10 score of 58.82%. We also see that, as opposed to logistic re­ gression and genetic algorithms, SVM is not suitable for the current data representation. After extensive experiments with SVMs, we still reach scores that are below baseline. In future work, we will investigate in more detail the limi­ tations of our re-ranking approach: which set of questions cannot be answered in the current system set-up and why?